Integrating Physical and Digital Interactions

T oday, physical and electronic media coexist like ships passing in the night: proximate but unaware of each other. Contemporary design studios, offices, and labs are filled with both physical and electronic artifacts, but the two exist separately, and the infrastructure for moving between media representations—scanning and print-ing—is heavyweight and cumbersome, at odds with the freewheeling, organic nature of creative work. For the past six years at UC Berkeley and Stanford University, my colleagues and I have conducted research into user interfaces that bind physical and electronic representations of artifacts for integrated interaction: Manipulation in one medium effects a corresponding change in the artifact's dual medium. Based on fieldwork with designers, office workers, scientists, and engineers, we've created and evaluated integrated interfaces ranging from whiteboards to oral history transcripts to field notebooks. We initially became interested in integrated interaction techniques while studying Web site designers. Mark Newman, James Landay, and I learned that, even in this high-tech industry, users preferred working with sticky notes and marking on walls and white-boards in a project's early design stages. This observation led us to create the Designers' Outpost Interactive whiteboard With this system, shown in Figure 1, users collaboratively author Web site architectures on a touch-sensitive electronic whiteboard augmented with cameras that capture documents placed on the wall. Designers create new pages by writing with standard pens on sticky notes and organize a Web site by physically moving the notes around on the board. The Designers' Outpost vision system uses two cameras to provide interactive feedback and near-interactive capture. A 640 × 480 pixel firewire video camera mounted inside the whiteboard offers interactive rate tracking of physical objects placed on the board. A high-resolution digital still camera captures the objects' contents , albeit at a lower frame rate—one frame every few seconds. Users can link notes by drawing a line with an electronic stylus as well as add ink annotations to a design to convey design rationale. Electronic capture enables replacing physical documents with their electronic images. Users have the same capabilities when content is electronic as when it's physical, thereby retaining physical direct manipulation. The main timeline is a visually navigable time-ordered set of design thumbnails. The thumbnails aren't just a visualization of the changes in the board—they're also a direct manipulation interface: Users can touch a thumbnail to return the board to the corresponding state. With Designers' Outpost, users interact with …